Neural Information Processing Systems http://nips.cc/

In vision-and-language grounding problems, fine-grained representations of the image are considered to be of paramount importance. Most of the current systems incorporate visual features and textual concepts as a sketch of an image. However, plainly inferred representations are usually undesirable in that they are composed of separate components, the relations of which are elusive. In this work, we aim at representing an image with a set of integrated visual regions and corresponding textual concepts, reflecting certain semantics. To this end, we build the Mutual Iterative Attention (MIA) module, which integrates correlated visual features and textual concepts, respectively, by aligning the two modalities. We evaluate the proposed approach on two representative vision-and-language grounding tasks, i.e., image captioning and visual question answering. In both tasks, the semantic-grounded image representations consistently boost the performance of the baseline models under all metrics across the board. The results demonstrate that our approach is effective and generalizes well to a wide range of models for image-related applications.

We explore those different aspects in relation to mutual knowledge, and analyze zero-shot predictions.

We explore those different aspects in relation to mutual knowledge, and analyze zero-shot predictions.

To close such semantic gap, we propose Concept Curation (CoCu), a pipeline that leverages CLIP to compensate for the missing semantics.

Neural Information Processing Systems http://nips.cc/

Table 2: The accuracy on the VQA v2.0 test set. We thank all the reviewers for the helpful comments. Q1: How the paper's contribution relates to the current SOT A? SGAE is a rather complicated scene-graph based method specific to image captioning. The results with current SOT A + MIA will be stated more clearly in the paper. Q2: How to use MIA on the baseline systems (i.e., how is MIA applied to image captioning For the settings, we have listed them in the supplementary materials.